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This project began in response to a perceived need to assess students perceptions with respect to the 
emerging use of audience response systems (clickers) in several mid- to large-size undergraduate 
courses at the University of Victoria. We developed and validated a “Clicker Use Survey” to gather 
students’ opinions with respect to clicker utility and the impact of clicker use on their learning. With 
the collected data we generated a set of baseline distributions to support assessment of various clicker 
use protocols and created a self-evaluation tool to share with instructors to support teacher reflection 
on the efficacy of their clicker practices. We also provided a sample self-evaluation to model the use 
of the tool. Links to the survey instrument, baseline data and self-evaluation tool, and sample self- 
evaluation are provided. 


Introduction 

C lassroom communication and interaction are 
influenced by many factors. In traditional large 
classes, communication tends to be between the 
teacher and a few keen and vocal students. The sheer 
number of students makes it infeasible to hear from 
every student and the traditional teacher-led ques¬ 
tion-answer process often fails to promote real dis¬ 
cussion. Many students choose to simply listen and 
passively take notes, relying on others to be the active 
participants in the class. This silent majority includes 
students who may remain silent because there are 
limited opportunities to contribute, lack confidence 


in their ability to articulate their ideas or questions, 
or avoid participation for fear of embarrassment or 
other social constraints (Dickman, 1993; Reynolds & 
Nunn, 1997; Pelton & Francis Pelton, 2003). Many 
of these students are on the verge of a teachable mo¬ 
ment - with connections ready to be made, misun¬ 
derstandings ready to be challenged, or interesting, 
unpopular or politically incorrect opinions needing 
to be discussed. By remaining silent, the students 
forfeit opportunities to learn and deny similar oppor¬ 
tunities to their classmates who may not be ready to 
articulate similar questions, ideas or concerns. 
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Reviewing existing research shows that tech¬ 
niques that promote and support student participa¬ 
tion yield students who pay more attention in class, 
use in-depth thinking and reflection, learn from one 
another’s reasoning and mistakes, evaluate their own 
misconceptions, perform better on tests, have great¬ 
er retention, and have improved attitudes toward 
learning (Francis Pelton & Pel ton, 2006). Audience 
response systems (also known as clickers’) were cre¬ 
ated to increase participation and engagement of au¬ 
dience members in large presentation halls or class¬ 
rooms where traditional teaching conditions might 
not support much interaction. The systems allow the 
instructor to present prepared or ad-hoc questions 
and response options to students and permit them 
to select and anonymously submit their responses 
through a wireless ‘clicker’ (typically radio frequency, 
but older systems also may use infra-red). Student 
responses are captured and processed in the instruc¬ 
tor’s laptop. The resulting distribution can then be 
presented graphically to the class via a data projector 
to support assessment, discussions, group activities, 
etc. Replacing or supplementing the traditional lec¬ 
ture and note-taking classes (passive learning) with 
this type of interactive process appears to support 
learning by increasing productive communication of 
processes, understandings and ideas and encourag¬ 
ing student engagement (Horowitz, 1988; Burnstein 
& Lederman, 2001; Guthrie & Carlin, 2004; Pelton 
and Francis Pelton, 2003; Pelton & Francis Pelton, 
2006). 

The Project 

The apparent benefits of using clickers in a classroom 
have encouraged both individual instructors and in¬ 
stitutions of higher education to introduce them in 
courses. This naturally leads instructors to question 
how their use of clickers impacts student learning. 
This project was designed by existing clicker users to 
help to more formally assess students’ attitudes to¬ 
wards the use of clickers in several mid to large size 
undergraduate courses. We developed and validated 
a “Clicker Use Survey” to assess students’ perceptions 
of clicker utility and the impact of clicker use on their 


learning and learning related behaviours, and pub¬ 
lished a collection of baseline distributions for the 
survey questions to support teacher self evaluation. 

The study context 

We collected data from 11 different mid- to large- 
enrollment undergraduate courses at the University 
of Victoria that were using clickers in their classes 
in the current year. Classes ranged in size from 40 
students to over 200 students. Classes covered a wide 
range of subjects, mainly in the sciences and social 
sciences (anatomy, anthropology, astronomy, chem¬ 
istry, computer science, geography, mathematics, and 
psychology). 

Two different audience response systems 
were used in the classes surveyed - the CPS (Class¬ 
room Performance System) by elnstruction and the 
iClicker. These systems were selected because they 
both used radio frequency communications (previ¬ 
ous experience had suggested that infra-red systems 
were not suitable for most large classroom situations) 
and they supported at least two operating systems (an 
important feature when seeking to make a campus 
wide recommendation). 

The elnstruction system had advantages with 
respect to included functionality but these advantag¬ 
es were tempered by a higher threshold (more time 
and effort required to learn) and friction (more time 
required to transform lessons and initiate use in class¬ 
rooms) as well as a higher cost or fee structure. The 
iClicker system was easier to set up and use, the costs 
were lower, and the software necessary to use the tool 
is open source (this potential for community based 
enhancement and customization was appealing to 
many instructors). 

Some instructors chose to assign marks to 
clicker use - either for participation or performance. 
Other instructors were only interested in using stu¬ 
dent responses to support student engagement in 
classroom activities and assess student understand¬ 
ing of the topics, and so, did not track individual 
participation. Originally we hoped that there might 
be some potential to examine the impact of the dif¬ 
ferences in response systems and differences in par¬ 
ticipation tracking on student perception of clicker 
utility, but other factors obscured the effects. 
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Methodology 

The Clicker Use Survey shown in Figure 1 includes 
questions on several demographic variables (gender, 
number of university courses completed, percentage 
of classes attended, and frequency of bringing the 
clicker to class), followed by 18 Likert type items as¬ 
sessing attitudes toward clickers and clicker utility. 
Students were also invited to make comments on 
anything pertaining to the clickers. These comments 
were explicitly encouraged in the verbal instructions 
given as the surveys were being distributed and again 
in print form on the bottom of the survey. The con¬ 
sent form also included a request to compare response 
patterns to student grades. 

The authors and participating instructors (all 
somewhat familiar with audience response technol¬ 
ogy) devised the items in the questionnaire to cover 
the apparent issues associated with the use of click¬ 
ers in the classroom. The development process was 
guided by informal classroom feedback, student eval¬ 
uations, an analysis of comments made on an anony¬ 
mous online forum (Slashdot, 2005), and experienc¬ 
es associated with the introduction and evaluation of 
other educational technologies. The instrument was 
piloted, item response distributions were assessed for 
functioning and suitability and minor revisions were 
made prior to administration to all classes. No at¬ 
tempt was made to control or influence the instruc¬ 
tors’ adoption or use of clickers in their classroom 
- leaving several variables unmanaged - but also 
providing a more authentic view into the issues sur¬ 
rounding the adoption of such devices. 

The responses were tallied for each of the lev¬ 
els in each Likert question in the survey. The order 
of the responses was reversed in negatively worded 
items to facilitate cross item comparisons. A cumu¬ 
lative ‘score’ or scale is not presented here because 
although we believe that we are examining evidence 
to support the validation of the instrument and the 
underlying latent construct (i.e., perceived clicker 
utility in the classroom), we do not believe that we 
have sufficient data to justify the assumption that all 
of the items would load perfectly onto a single scale 
or that values generated would fall on a linear scale. 


The following sections provide a short sum¬ 
mary of the observed response distributions and a 
qualitative analysis and interpretation along with the 
results of a simple factor analysis (suspending our 
concerns about the aggregation of non-linear likert 
data). We also present a tool (Pelton, Sanseverino, & 
Francis Pelton, 2007) to support instructors wishing 
to apply the “Clicker Use Survey” to qualitatively 
evaluate the efficacy and utility of clickers in their 
classes. Future papers will examine relationships be¬ 
tween demographics, grades, and student responses 
(both Likert and open-ended). 

Results 

The Clicker Use Survey was presented to the 11 class¬ 
es at the end of their respective terms in an 8 month 
period. Informal observations suggested that more 
than 90% of the students attending classes on the 
days that the survey was administered consented to 
the study and completed the questionnaire. A total 
of 1107 questionnaires were completed, representing 
approximately 70% of the students enrolled in the 
participating first- and second-year classes. 

Four conditions were anticipated to affect 
student assessment of clicker use. The teaching style 
was expected to have a substantial effect and while we 
cannot control this effect, it is hoped that the base¬ 
line distributions resulting from this research will al¬ 
low instructors to self-evaluate the response distribu¬ 
tions from their classes in juxtaposition to a ‘typical’ 
response distribution. A second expected influence 
on perceived utility is the use of grades or marks to 
encourage or coerce students into purchasing (if nec¬ 
essary), registering, and using the clickers in the class¬ 
room. Third, the specific system used was expected to 
have a minor effect on student perception of response 
systems because of the differences in registration, dai¬ 
ly setup, and fluidity. And finally we anticipated that 
there would be a negative effect on perceived clicker 
use if the students were required to purchase their 
own clicker units. However, once the data was col¬ 
lected and the conditions in the classrooms examined 
we found that all of the classes that used the tracked 
model, and all of the classes that offloaded equip¬ 
ment costs to students, used the elnstruction system. 
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Figure i 

Clicker Use Survey 

Clicker Use Survey 

To evaluate the effectiveness of clickers, it is necessary to ask for honest responses from students using the devices. 
Your co-operation is very much appreciated. Your responses are confidential and only anonymous comments and 
aggregate results will be disclosed. 

Student ID:_ Check One: Male _ Female_ 

Flow many units (1 course = 1.5 units) of University level courses have you taken (including this term)? _ 

What percentage of classes in this course did you attend? _0-25% _26-50% _51-75% _76-100% 

How often did you bring the clicker to class: _Never_Sometimes _Most of the time _Always 

Please respond to each question by circling one number from 1 to 5. 


Strongly Disagree Undecided Agree Strongly 
Disagree Agree 


1. Class time passes more quickly when we 
use the clickers. 

1 

2 

3 

4 

5 

2. When we use the clickers my participation 
increases in other ways too. 

1 

2 

3 

4 

5 

3. I found the use of clickers to be 
distracting and unhelpful. 

1 

2 

3 

4 

5 

4. I feel uncomfortable sharing my responses 
via the clickers. 

1 

2 

3 

4 

5 

5. Learning with clickers improves my 
understanding of the course content. 

1 

2 

3 

4 

5 

6. Learning with the clickers gives me 
confidence to ask more questions. 

1 

2 

3 

4 

5 

7. Using the clickers encourages me to spend 
more time preparing for class. 

1 

2 

3 

4 

5 

8. Using clickers encourages me to attend 
more classes. 

1 

2 

3 

4 

5 

9. Using the clickers promotes more focused 
discussion during class. 

1 

2 

3 

4 

5 

10. I would like to use the clickers in other 

courses. 

1 

2 

3 

4 

5 

11. The graphs provided by the clicker system 
are useful. 

1 

2 

3 

4 

5 

12. I benefit by seeing how other students 
respond to a question. 

1 

2 

3 

4 

5 

13. Using the clickers in class is too time 
consuming. 

1 

2 

3 

4 

5 

14. I would do better in this class without the 
clickers. 

1 

2 

3 

4 

5 

15. I had difficulties getting my clicker to 
work in class. 

1 

2 

3 

4 

5 

16. I would have liked to use the clickers 
more often in class. 

1 

2 

3 

4 

5 

17. At first, learning with the clickers was 
enjoyable but later I was bored. 

1 

2 

3 

4 

5 

18. Using the clickers helped to better prepare 
me for quizzes and exams. 

1 

2 

3 

4 

5 


Please write any comments you would like to make on the back of this form. 


© 2006, Francis Pelton, Pelton, and Sanseverino 
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In addition, it was observed that the instructors us¬ 
ing the tracked model tended to have fewer years of 
teaching experience. Thus, three of our conditions 
were confounded and so the data does not support a 
full exploration of all of our intended comparisons of 
conditions. However, the aggregated response distri¬ 
butions for the survey questions were still perceived 
as useful both for the validation of our instrument 
and as a baseline of perceived clicker utility to iden¬ 
tify trends examine interventions and support self- 
evaluation. 

To provide an example of the results collect¬ 
ed from the survey, a selection of class distributions, 
as well as a baseline distribution (all students) for 
question #16, “I would have liked to use the click¬ 
ers more often in class,” is presented in Figure 2. In 
the baseline distribution (all students, front, light 
blue bars), we see that across all classes, the students 
were generally content with the level of clicker use. 
For some classes (e.g., 1 and 4), the response distri¬ 
bution includes a substantially larger “strongly dis¬ 
agree” component than the baseline. Possible rea¬ 
sons might include, overuse of clicker questions in 
classroom, the instructors’ skill in using clickers in 
the classroom, or negative grading affected students’ 
desire to use clickers (both of these classes associated 
grades with clicker use). 

Various facets affecting students’ perceptions 
of clicker use can be assessed using the collected data. 
For example, in Figure 3, we compare response distri¬ 
butions from classes where participation was tracked 
and marks were associated with clicker use and the 
other classes that did not link marks to clicker re¬ 
sponses or participation. We have reversed all nega¬ 
tively oriented questions to facilitate evaluation and 
comparison across items - 1 indicates a very negative 
attitude or opinion and a 5 indicates a very positive 
attitude or opinion. Note that while all tracked class¬ 
es were using the elnstruction system, both systems 
were used in untracked classes. In addition, tracked 
classes tended to have instructors with fewer years of 
teaching experience. The baseline combines all class¬ 
es and we see that the student response distributions 
from the tracked classes were substantially negatively 
affected. Possible reasons for the observed differences 
in the distribution include experience of the instruc¬ 


tor or negative attitudes in response to tracking par¬ 
ticipation. 

As we reviewed each of the survey item re¬ 
sponse distributions, we noted that the baseline 
distribution provided a useful starting point for the 
evaluation and interpretation of group and individu¬ 
al class results. For some questions, the variation ob¬ 
served across groups or classes was minimal, while for 
others the class effect was substantial. Other papers 
examining the distributions of each question and the 
student comments are in progress. 

Some general observations arising from our 
examination of the response distributions include: 

• students generally felt comfortable sharing 
responses; 

• students agreed that seeing the response dis¬ 
tributions in class was beneficial; 

• class distributions were less positive when 
marks were associated with clicker use; and 

• clicker use did not seem to yield substantial 
increases in student preparation or confi¬ 
dence to ask questions in class. 

These observations were consistent with a cursory re¬ 
view of student comments in connection with the 
Likert question response patterns. 

We began our factor analysis by including all 
items and found that perhaps two or three factors 
might be present in the raw data. However upon 
reflecting on our search for the intended factor of 
clicker utility we decided to remove questions 4, 7, 8, 
15 and 18 - as they spoke most directly to other is¬ 
sues such as personal comfort (4), learning habits (7, 
8), technological struggles (15) and format of class 
assessments (18). The second factor analysis provided 
moderate evidence to assert a single factor (46.8% 
of the variance accounted for in the first factor and 
the screen plot suggested a single factor). This factor 
analysis evidence in concert with the examination of 
response distributions, the iterative item construction 
process using experienced clicker users, the piloting 
and refining process, and the consistency of observa¬ 
tion distributions with classroom observations and 
classroom descriptions suggests that the Clicker Use 
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Figure 2 

Selected class distributions for survey question #16: 

"I would have liked to use the clickers more often in class." 



Rating 


Survey can be used to meaningfully assess the utility 
of clickers in individual classrooms. 

Self Evaluation Tool 

The Clicker Use Survey is freely available to the aca¬ 
demic community to support institutional and self- 
evaluation of classroom clicker use. 

The authors have created a self-evaluation 
tool to support instructors wishing to examine stu¬ 
dent perceptions of clicker utility as adopted in their 
own classrooms. This resource has been published in 
the form of an Excel spreadsheet (Pelton, Francis Pel- 
ton, & Sanseverino, 2007) that contains the baseline 
survey response distributions for 1107 students along 
with preset data entry fields for the instructor to en¬ 
ter his/her class response distributions. Comparative 
distribution graphs are automatically generated once 


the instructor has entered a set of class data, and these 
can be copied and used in self-evaluation reports. 

Finally, an exemplar self-evaluation report 
(Francis Pelton & Pelton, 2007) is provided as a 
model to help other instructors generate supporting 
documentation for their own teaching dossiers. 

The Clicker Use Survey was designed to as¬ 
sess student perceptions of the use and utility of 
clickers in the classroom. The items were generated, 
piloted, refined, and evaluated by instructors familiar 
with the use of audience response systems. When this 
response system expertise is combined with the in¬ 
tuitively interpretable baseline data and distributions 
and the adequate loading of the items on a single fac¬ 
tor (given the inherently multifaceted nature of the 
classroom context), it provides adequate validation 
evidence to support the practical application of this 
instrument in assessing the perceived utility of click¬ 
ers in the classroom, the examination of the relative 
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Figure 3 

Tracked vs. untracked distributions for survey question #14: 
"I would do better in this class without the clickers." 



efficacy of various clicker related protocols, and the 
examination of changes over time in the instructors 
ability to effectively use clickers in their classroom. 
The authors welcome comments, queries, and sug¬ 
gestions on this resource from other educators. 
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